The number of international benchmarking competitions is steadily increasing in various fields of machine learning (ML) research and practice. So far, however, little is known about the common practice as well as bottlenecks faced by the community in tackling the research questions posed. To shed light on the status quo of algorithm development in the specific field of biomedical imaging analysis, we designed an international survey that was issued to all participants of challenges conducted in conjunction with the IEEE ISBI 2021 and MICCAI 2021 conferences (80 competitions in total). The survey covered participants' expertise and working environments, their chosen strategies, as well as algorithm characteristics. A median of 72% challenge participants took part in the survey. According to our results, knowledge exchange was the primary incentive (70%) for participation, while the reception of prize money played only a minor role (16%). While a median of 80 working hours was spent on method development, a large portion of participants stated that they did not have enough time for method development (32%). 25% perceived the infrastructure to be a bottleneck. Overall, 94% of all solutions were deep learning-based. Of these, 84% were based on standard architectures. 43% of the respondents reported that the data samples (e.g., images) were too large to be processed at once. This was most commonly addressed by patch-based training (69%), downsampling (37%), and solving 3D analysis tasks as a series of 2D tasks. K-fold cross-validation on the training set was performed by only 37% of the participants and only 50% of the participants performed ensembling based on multiple identical models (61%) or heterogeneous models (39%). 48% of the respondents applied postprocessing steps.

生物医学图像分析算法验证取决于参考数据集的高质量注释，标记指令是关键。尽管它们的重要性，但他们的优化仍然没有得到探索。在这里，我们介绍了对标签指令及其对该领域注释质量的影响的首次系统研究。通过对Miccai协会注册的专业实践和国际比赛的全面检查，我们发现了注释者对标签说明的标签需求及其当前质量和可用性之间的差异。基于对156家专业公司的156个注释者和708个亚马逊机械土耳其人（MTURK）人群的注释者的14040张图像的分析，使用具有不同信息密度级别的说明，我们进一步发现，包括示例性图像与文本描述，唯一的描述，示例性图像显着增强了注释性能，虽然仅扩展文本说明并非如此。最后，专业注释者不断优于mturk人群。我们的研究提高了对生物医学图像分析标签指令中质量标准的需求的认识。